Moments and \(L^p\) spaces
Moments
Let \(X:\Omega\to\R\) be a random variable defined on the probability space \((\Omega,\sF,P)\) with the distribution function \(F_X:\R \to [0,1]\).
Example 1 (Absolute value function). For the function \(\abs{\cdot}: \R \to \R_+\), we can write the inverse image of half open sets \((-\infty, x]\) for any \(x \in \R\) as \(A_g(x) = g^{-1}(-\infty, x]\). It follows that \(A_g(x) = \emptyset \in \sB(\R)\) for \(x < 0\) and \(A_g(x) = [-x, x] \in \sB(\R)\) for \(x \in \R_+\). Since \(g^{-1}(-\infty, x] \in \sB(R)\), it follows that \(\abs{\cdot}: \R \to \R_+\) is a Borel measurable function.
Lemma 2. If \(\E\abs{X}\) is finite, then \(\E X\) exists and is finite.
Proof. Proof. The function \(\abs{\cdot}: \R \to \R\) is a Borel measurable function and hence \(\abs{X}\) is a random variable. Further \(\abs{X} \ge 0\), and hence the expectation \(\E\abs{X}\) always exists. If \(\E\abs{X}\) is finite, it means \(\E X_+\) and \(\E X_-\) are both finite, and hence \(\E X = \E X_+ - \E X_-\) is finite as well. ◻
Corollary 3. Let \(g: \R \to \R\) be a Borel measurable function. If \(\E\abs{g(X)}\) is finite, then \(\E g(X)\) exists and is finite.
Exercise 4 (Polynomial function). For any \(k \in \N\), we define functions \(g_k: \R \to \R\) such that \(g_k: x \mapsto x^k\). Show that \(g_k\) is Borel measurable for all \(k \in \N\).
Definition 5 (Moments). We define the \(k\)th moment of the random variable \(X\) as \(m_k \triangleq \E g_k(X) = \E X^k\). First moment \(\E X\) is called the mean of the random variable.
Remark 1. If \(\E\abs{X}^k\) is finite, then \(m_k\) exists and is finite.
Remark 2. If \(P\set{\abs{X} \le 1} = 1\), then \(P\set{\abs{X}^k \le 1} = 1\). Therefore, by the monotonicity of expectations \(\E\abs{X}^k \le 1\), and the moments \(m_k\) exist and are finite for all \(k \in \N\).
\(L^p\) spaces
Definition 6. A vector space over field \(\F\) is denoted by a set \(V\) has (a) vector addition \(+: V\times V\to V\) that satisfies associativity and commutativity, and has an identity and inverse element for each vector \(v\in V\), and (b) scalar multiplication \(\cdot: \F \times V \to V\) that satisfies compatibility of field multiplication, distributivity with vector and field addition, and has an identity element.
Remark 3. We can verify that the set of random variables is a vector space over reals, and we denote it by \(V\). To this end, we can verify the associativity and commutativity of addition of random variables. For any random variable \(X\), the constant function \(0\) is the identity random variable, and \(-X\) is the additive inverse for vector additions. We can verify the compatibility of scalar multiplication with random variables, existence of unit scalar \(1\), and distributivity of scalar multiplications with respect to field addition and vector addition.
Definition 7. For a vector space \(V\) over field \(\F\), a norm on the vector space is a map \(f: V \to \R_+\) that satisfies
\(f(av) = \abs{a}f(v)\) for all \(a \in \F\) and \(v \in V\),
\(f(v+w) \le f(v) + f(w)\) for all \(v,w \in V\), and
\(f(v) \ge 0\) for all \(v \in V\).
Definition 8. For a probability space \((\Omega,\sF,P)\), and \(p \ge 1\), we define the set of random variables with finite absolute \(p\)th moment as the vector space \(L^p \triangleq \set{X \in V:(\E\abs{X}^p)^{\frac{1}{p}} < \infty}.\)
Definition 9. We define a function \(\norm{}_p: L^p \to \R_+\) defined by \(\norm{}_p(X) = \norm{X}_p \triangleq (\E\abs{X}^p)^{\frac{1}{p}}\) for any \(X \in L^p\) and real \(p \ge 1\).
Remark 4. For \(p=1\), the map \(\norm{}_p\) is norm. Therefore, \(L^1\) is a normed vector space consisting of random variables with bounded absolute mean.
Remark 5. We will show that \(\norm{X}_\infty = \sup\set{\abs{X(\omega)}:\omega\in \Omega}\), and hence \(L^\infty\) is a normed vector space of bounded random variables.
Remark 6. We will also show that \(\norm{}_p\) is a norm for all \(p \in (1, \infty)\), and hence \(L^p\) is a normed vector space of random variables with bounded \(\norm{}_p\) norm. In particular, the \(L^2\) space consists of random variables with bounded second moment.
Remark 7. If \(\E\abs{X}^N\) is finite for some \(N \in \N\), then \(\E\abs{X}^k\) is finite for all \(k \in [N]\). This follows from the linearity and monotonicity of expectations, and the fact that This implies that \(L^N \subseteq L^k\) for all \(k \in [N]\). We will show that \(L^{q} \subseteq L^p\) for any real numbers \(1 \le p \le q\).
Central Moments
Example 10 (Shifted polynomial functions). For any \(k \in \N\), we define functions \(h_k: \R \to \R\) such that \(h_k: x \mapsto (x-m_1)^k\). Then, \(h_k = g_k(x-m_1) = g_k\circ f\) where \(f:\R \to\R\) is defined as \(f(x) = x-m_1\) for all \(x \in \R\). Since \(g_k\) and \(f\) are measurable, so is \(h_k\).
Definition 11 (Central moments). Let \(X: \Omega \to \R\) be a random variable defined on the probability space \((\Omega, \sF, P)\) with finite first moment \(m_1\). We define the \(k\)th central moment of the random variable \(X\) as \(\sigma_k \triangleq \E h_k(X) = \E (X-m_1)^k\). The second central moment \(\sigma_2 = \E(X-m_1)^2\) is called the variance of the random variable and denoted by \(\sigma^2\).
Lemma 12. The first central moment \(\sigma_1= \E(X-m_1) = 0\) and the variance \(\sigma^2 = \E(X-m_1)^2\) for a random variable \(X\) is always non-negative, with equality when \(X\) is a constant. That is, \(m_2 \ge m_1^2\) with equality when \(X\) is a constant.
Proof. Proof. Recall that \(h_1, h_2\) are Boreal measurable functions, and hence \(h_1(X) = X- m_1\) and \(h_2(X) = (X-m_1)^2\) are random variables. From the linearity of expectations, it follows that \(\sigma_1 = \E h_1(X) = \E X - m_1 = 0\). Since \((X-m_1)^2 \ge 0\) almost surely, it follows from the monotonicity of expectation that \(0 \le \E(X-m_1)^2\). From the linearity of expectation and expansion of \((X-m_1)^2\), we get \(\sigma^2 = \E X^2 - 2m_1\E X + m_1^2 = m_2 - m_1^2 \ge 0.\) ◻
Remark 8. If second moment is finite, then the first moment is finite. That is, \(L^2 \subseteq L^1\).
Inequalities
Theorem 13 (Markov’s inequality). Let \(X: \Omega \to \R\) be a random variable defined on the probability space \((\Omega, \sF, P)\). Then, for any monotonically non-decreasing function \(f: \R \to \R_+\), we have
Proof. Proof. We can verify that any monotonically non-decreasing function \(f: \R \to \R_+\) is Borel measurable. Hence, \(f(X)\) is a random variable for any random variable \(X\). Therefore, The result follows from the monotonicity of expectations. ◻
Corollary 14 (Markov). Let \(X\) be a non-negative random variable, then \(P\set{X \ge \epsilon} \le \frac{\E X}{\epsilon}\) for all \(\epsilon > 0\).
Corollary 15 (Chebychev). Let \(X\) be a random variable with finite mean \(m_1\) and variance \(\sigma^2\), then
Proof. Proof. Apply the Markov’s inequality for random variable \(Y = \abs{X-m_1} \ge 0\) and increasing function \(f(x) = x^2\) for \(x \ge 0\). ◻
Corollary 16 (Chernoff). Let \(X\) be a random variable with finite \(\E[e^{\theta X}]\) for some \(\theta>0\), then
Proof. Proof. Apply the Markov’s inequality for random variable \(X\) and increasing function \(f(x) = e^{\theta x} > 0\) for \(\theta > 0\). ◻
Definition 17 (Convex function). A real-valued function \(f: \R \to \R\) is convex if for all \(x, y\in \R\) and \(\theta \in [0,1]\), we have
Theorem 18 (Jensen’s inequality). For any convex function \(f: \R \to \R\) and random variable \(X\), we have
Proof. Proof. It suffices to show this for simple random variables \(X: \Omega \to \sX\). We show this by induction on cardinality of alphabet \(\sX\). The inequality is trivially true for \(\abs{\sX}=1\). We assume that the inductive hypothesis is true for \(\abs{\sX} = n\).
Let \(X \in \sX\), where \(\abs{\sX} = n+1\). We can denote \(\sX = \set{x_1, \dots, x_{n+1}}\) with \(p_i \triangleq P\set{X = x_i}\) for all \(i \in [n+1]\). We observe that \((\frac{p_j}{1-p_1}: j \ge 2)\) is a probability mass function for some random variable \(Y \in \sY = \set{x_2, \dots, x_{n+1}}\) with cardinality \(n\). Hence, by inductive hypothesis, we have Applying the convexity of \(f\) to \(\theta = p_1, x = x_1, y = \sum_{i=2}^{n+1}\frac{p_i}{1-p_1}x_i\), we get From the inductive step, it follows that the RHS is upper bounded by \(\E f(X)\), and the result follows. ◻
Theorem 19. For any real numbers \(1 \le p \le q < \infty\), we have \(L^q \subseteq L^p\).
Proof. Proof. Let \(1 \le p \le q < \infty\) and consider a convex function \(g: \R_+ \to \R_+\) defined by \(g(x) \triangleq x^{{q}/{p}}\) for all \(x \in \R_+\). It follows that \(g(\abs{X}^p) = \abs{X}^q\) and hence from the Jensen’s inequality, we get ◻